Deploy and Use Open Source GPT Models for RAG

You should see the list of models.

You should now see llama3.1 as the selected model.

You will see a green banner informing you the default model was updated.

In a real environment, the green banner disappears after a few seconds. You should now see a clean chat page.

Notice how the model stayed the same, but the prompt suggestions changed as expected.

In the next couple of steps, you will learn how to refer to the documents when prompting. The OpenWeb UI uses the # symbol followed by the collection or filename to associate the prompt with the desired knowledge base.

You should see all the collections and files in a drop-down menu.

You should see the collection in the chat window.

The following prompt will be associated with the collection containing the PDF of the entire interface configuration guide.

You should see the prompt inserted in the chat field.

You should see the response.

The answer you were looking for was supposed to be more like a "Yes" or "No" in nature. Looking at the documentation yourself, you find out that the answer is a clear "no". You can see in the Note on page 131 that you cannot configure static MAC address on tunnel interfaces.

You will examine the output and database entries used by llama3.1 while generating the answer to determine why it provided an incorrect response.

Notice how it mentions the part of the document referring to the words Layer 3 and static MAC addresses. It also asks you to clarify what kind of interface you had in mind.

You will not change the prompt as suggested. First you will explore what kind of entries from the database the system actually used when generating the answer.

You should see the Citation page displaying the entries from the database that were used as context.

The Source field shows the exact file that was used for this citation with the page in the parenthesis. You can see it took the text from page 404.

Next is the Relevance score. The higher the score, the better the semantic match with the prompt. A score of 27.96 % is considered quite low. This entry is considered the best of the overall top 5 entries in the database.

Examining the entry, you can see that it is almost impossible for a human to read. Looking at this text and comparing it to the actual PDF, and using some imagination, you can figure out that it actually took the text from page 405.

You should consider the page count as a +/- 1 ballpark figure.

Notice how many times the word tunnel appears in the citation. The prompt was asking about tunnel interfaces and the system calculated semantic similarity based on the used words. The repetition of the word tunnel is why the system chose this particular entry as the most relevant context. The system determined that it is statistically most likely to match. Notice also that this text is part of some configuration syntax.

You should see the second-best match.

This match has a very similar similarity score as the first one. Again you can see that it matched the configuration syntax due to the repetition of the word tunnel.

You can optimize the entire RAG process by grouping the data into more meaningful parts yourself. The easiest way is to isolate the most relevant chapter and use it as a collection.

There is another collection that contains only the chapter about Layer 3 interface configuration. You will use this reduced knowledge base to see if the answer improves.

You will return to the chat interface.

You should see a fresh chat interface.

You should see the files and collections menu.

You should see the collection added to the prompt field.

Keep in mind that this is only one chapter from the entire configuration guide.

You should see the prompt added in the field.

You should see an answer.

This time the answer is correct and direct. A tell-tale sign of GPT hallucination is unnecessary verbosity. Now you will explore the citations to verify whether the answer was really based on the retrieved context.

You should see the best match.

Notice that the relevance score is 25.74 %, which is quite lower compared to the relevance score from the previous prompt, which was around 29 %. You can also see that this text chunk contains the note where it is explicitly written you cannot configure a static MAC address on a tunnel interface.

RAG works by retrieving information from a dataset and using it to generate answers. However, if the dataset is too large or contains a lot of irrelevant information (noise), it can confuse the system and lead to poor results. This is why understanding your dataset is so important. If you know your data well, you can create better, more focused prompts that guide the system toward the right information.

It also helps to prepare your data in ways that make it easier for RAG to work with. For example, instead of breaking the data into fixed-sized chunks, you can use semantic chunking. This means splitting the data into smaller pieces based on meaning, so each chunk contains complete and useful information. For example, in a PDF, one chapter might begin discussing VLANs, while later chapters delve into more detailed discussions of VLANs. Semantic chunking helps by grouping these disparate sections into a cohesive and meaningful piece of text. This increases the likelihood of RAG retrieving the most relevant information.

In the following task, you will explore some prompt engineering techniques and refine your queries to get the most out of a RAG system.

You should see a clean chat interface.

You should see the inserted prompt.

You should see the answer.

It seems like the system did not check if VLAN 5010 is even a valid VLAN ID. It took the prompt verbatim and started to provide step-by-step instructions on how to configure a VLAN, which was 5010 in this case. In other words, asking can you do something is not the same as asking whether it is possible to do something. Even so, the system provided factually incorrect answers. Remember to always fact-check the answers. Your network engineering expertise becomes crucial in this case. In other words, GPT and RAG are merely tools that heavily depend on the user. By no means are these tools a replacement for network engineers and their expertise.

First, Cisco Nexus 9000 switches have only one single VDC so this first part of the answer is irrelevant. The VDC feature is used on Cisco Nexus 7000 switches. Second, the show vdc all command is invalid, since there is no all flag for this command.

You should see the next part of the answer.

Here you also see another problem. The feature vlan command does not exist on the Cisco Nexus 9300.

You should see the final part of the answer.

The last part tells you how to configure an interface as a trunk and allow this single VLAN on the trunk port. The configuration is correct, but the GPT system did not tell you about the option of configuring an access port with this VLAN.

You will try again, this time using a different prompt.

You should see a new chat.

The prompt is now inserted in the chat field.

You should see the answer.

This time the answer is correct. A small change in the prompt provided you with a completely different answer.

How different the answers to the same or a similar prompt will be is defined also by a parameter called temperature. You can also influence the answers by providing additional instructions in the prompt itself. You will explore the effect of the temperature parameter and adding the instructions in the following steps.

You should see a new chat interface.

You should see the following prompt inserted: You are a Cisco network expert. Provide short and direct answers. Is it allowed to configure VLAN 5010 on an interface of a Cisco Nexus 9000 series switch?

Notice the instructions to provide short and direct answers.

You should get a succinct answer. It is correct also this time around.

The instructions worked. Now you will explore the temperature parameter and its effects.

It is located in the top-right corner of the chat interface.

You should see the chat settings side-panel.

For now, focus only on the temperature parameter. All these settings are meant for advanced tailoring of how the GPT responds and is not covered in this course.

You should see the temperature slider set to the default value of 0.8.

The temperature parameter ranges typically from 0 to 1 so 0.8 is considered very high. Higher temperature values allow for more expressive and creative answers, whereas lower values are used to get more basic and repetitive responses.

This sends the same prompt and generates a different answer.

The answer is somewhat correct. It answered correctly regarding 5010 VLAN ID being out of range but notice how it incorrectly defined numerical ranges. This kind of mistakes regarding numbers are very common in GPT-generated answers. Always fact check the answers that you receive.

Higher temperature settings can lead to more elaborate and erroneous answers. You will lower the temperature and regenerate the answer in the following steps.

This action sets it to 0.2.

You should see the answer that was generated with the lowered temperature value.

Notice how the answer is much simpler, direct, and correct. Note, that lowering the temperature only increases the likelihood of getting a correct answer but does not guarantee it, so always fact check the answers.

You should get a very similar answer.

Lower temperature settings also provide more similar answers when entering the same or similar prompts.

Now that you have learned the importance of prompt engineering and temperature, you will test the RAG system with the Fabric Information collection containing running configurations of the switches and various outputs from show commands.

You should see a clean interface with all the default settings.

You should see the Fabric Information collection in the drop-down menu.

You should see the collection in the prompt field.

You should see the following prompt: List all management interfaces IP addresses found in the Fabric Information collection.

You should see the generated answer.

The retrieved IP for spine01 is correct but the system failed to tell you about the other two devices. Inspect the citations to figure out why it did not produce IPs for all devices and to verify if the retrieved IP is really valid and originates from the documentation.

You should see the citation.

You can see that the retrieved IP is indeed from the running configuration. If you checked the citation of leaf02_running_config.txt file, you would see the text in the following figure.

This part of the configuration has no mentions of the management VRF or IP. Interestingly, the relevance score is almost the same. Remember, a RAG system uses your prompt to look for similar entries in the database. You can try to get better results by iterating with the GPT.

Prompt iteration is a technique where you correct the GPT and provide more detailed information to get a better answer. It looks like a dialogue.

You will return to the chat interface.

You should see the following prompt added: No, I need you to list all management IP addresses for leaf01, leaf02 and spine01 devices.

Notice how the entire chat interface looks like a messenger app with two people talking.

The goal of this prompt is to tell the GPT the answer was not what you expected and to provide more specific details, this time including the device names. Note that each TXT file is named after the device and that the first line includes information regarding the device name and the used command that produced the output.

You should see the following answer.

This time you did get three distinct IP addresses. Time to verify if they really are the management IP addresses. Note that this fabric has a BGP underlay with an EVPN overlay, which means, there are several IP addresses configured. Always fact-check the answers from a GPT system, even when using RAG.

You should see the output for the management VRF of leaf02.

Notice how the IP address is very close to the mgmt0 keyword. This output was gathered using the following commands: show ip route vrf management, show bgp vrf management all, show ip arp vrf management, show ip interface vrf management. The citation you see here is only one small part of the entire leaf02_management_vrf_data.txt file.

Keep in mind that you were using a fixed-length chunk size of 500 characters with a 100-character overlap. In such cases, it is often beneficial to provide the same information in different formats. For instance, running configuration and VRF data.

You will return to the chat interface.

You should see the best match for leaf01.

Notice that the exact same part of the output is chosen, only this time for leaf01.

You will return to the chat interface.

You should now see the rest of the citations.

You should see the window move to the right, revealing the other citations in full.

The spine01_management_vrf_data.txt contains the same part of the file, as you can see in the figure.

The output of the show commands for the management VRF turned out to be the best match for this prompt. This showcases the importance of careful prompt engineering and data preparation with RAG systems. What about the other two remaining citations?

You should see the output of the show cdp neighbors command for leaf02.

This citation is irrelevant and can be treated as unnecessary noise. The same goes for the leaf01_cdp_neighbors.txt citation. To address this issue, you can improve the process by using semantic chunking techniques or integrating a re-ranker. Semantic chunking ensures that the text is divided into meaningful segments that better align with the context of the prompt. By optimizing your chunking you can prioritize splitting based on semantic boundaries such as topic shifts or keyword density rather than arbitrary text lengths. This helps isolate the most relevant information while reducing the inclusion of irrelevant noise.

A re-ranker can further refine the results by evaluating and prioritizing the most contextually relevant chunks. By using a more advanced embedding model or a pre-trained re-ranker specialized in contextual understanding, you can filter out citations like leaf01_cdp_neighbors.txt, which do not add meaningful value to the context. This ensures that only the most relevant chunks are passed to the GPT model.

Since GPT processes all matched citations as a single context, minimizing unnecessary or noisy citations is crucial to avoid diluting the model's focus. Combining semantic chunking with a re-ranker can significantly enhance the quality of the input context, leading to more precise and accurate responses tailored to the prompt.

Satisfied with the capabilities of your on-premises small-scale RAG deployment, you use all that you learned during this testing to boost your productivity and optimize tedious, mundane, and error-prone tasks, supercharging your day-to-day work.